The Role of Multimodal in the Learning Sensorimotor Involvement of Cognitive Skills

The existence and extent of effects of small-scale haptic involvement on recall performance in an explicit memory task were investigated in two related experiments involving a total of 40 subjects. The memory task was structured in such a way that it abstracts two of the fundamental activities encountered when interacting with realworld equipment through physical control interfaces: 1) the identification of the state of the system to be acted on; and 2) the recall of the series of actions which are to be taken when the system is in that state. Furthermore, the experiments were designed as training experiments. Haptic sensorimotor involvement is evaluated as an aid to training -it is used during practice and not used during performance. Results from Memory Experiment 1 show (i.e., Ho: p < 0.01 for 5 minute, 30 minute, and 24 hour examinations) that experimental group subjects performed approximately 2.5 times better when they trained with both visual stimuli (i.e., a keypad graphic) and haptic involvement than when they trained without these visual and haptic stimuli. Memory Experiment 2 shows (i.e., Ho: p < 0.05 for 5 and 30 minute examinations) that experimental group subjects performed approximately 1.5 times better when they trained with the keypad graphic and haptic interaction than when they trained with the keypad graphic only. These results were obtained even though the design of the experiments was such that independent variables affected the training of only the action part of sequences (i.e., the training of the sequence label was not affected by the independent variables) and experimental group examinations required recall of sequences memorized under two different conditions. The results of the two memory experiments suggest that: 1) small-scale haptic involvement can act as a training aid for a purely cognitive task; and 2) small-scale haptic involvement can compliment visual stimuli as an aid to training in a purely cognitive task. Thesis Supervisor : Nathaniel I. Durlach Title: Senior Scientist, Department of Electrical Engineering and Computer 1.0 Introduction The increased commercial availability of interface and computational devices supporting the development of multimodal, spatially-oriented, interactive human/machine interfaces has contributed to a burgeoning interest in the field of human/computer interaction (Biocca, 1992). Devices such as binocular stereoscopic helmet mounted displays, binaural stereophonic spatialization devices, haptic interaction devices, and unobtrusive position/orientation sensors are enabling the creation of human/machine interfaces for interaction with telerobots and computational models of a form and type which were previously found only in specialized research and military efforts (e.g., Freund, 1986; Shaker and Wise, 1988; Tachi, Arai, and Maeda, 1989; Aviles, Hughes, Everett, et. al., 1990). In their most extreme manifestations, systems utilizing these technologies are attempting to remove the boundaries associated with devices mediating interaction between humans and remote task environments (physical and computer generated) and, in some sense, project the human into these remote or virtual environments. Virtual environment (VE) systems in particular are experiencing a high degree of interest from the military, commercial, and private sectors. Current and foreseen application domains for VE systems include: (1) design, manufacturing, and marketing; (2) medicine and health care; (3) entertainment; and (4) training (Durlach and Mavor, 1995). This discussion will focus on research issues associated with using VE systems and technology for training. From a training perspective, some key features of virtual environments are that they are: 1) interactive and adaptive; 2) reconfigurable in software; 3) multimodal; and 4) can generate supernormal situations (Durlach, Pew, Aviles, DiZio, and Zeltzer, 1992). These features allow VE-based training systems to be developed which can be precisely tailored to specific tasks and individuals. In many ways, however, VE-based training systems may be viewed as natural outgrowths of previous simulation-based training systems (e.g., flight trainers). In some form, all four aforementioned key VE features apply to these "classical" trainers. In what ways, therefore, are VE-based systems unique? First of all, in contrast to classical simulation systems, VE-based systems can be highly reconfigurable for BOTH the far-field (i.e., objects and interactions out of reach) and the near-field (i.e., objects and interactions within reach). Classical flight simulators, for example, only reconfigure the scenery and environment outside of the airplane (i.e., the far-field) and do not easily allow flexibility in the airplane display and control layout (i.e., the near-field). In contrast, a VE-based training system can simulate a variety of nearand far-field configurations while utilizing the same human/machine interface devices. The second key characteristic of VE-based training systems is the potential for allowing multimodal (i.e., visual, auditory, and haptic) interactions. Unlike classical computer-based simulation systems, near-field simulations can be generated which may seen, felt, and heard in a spatially-oriented manner. Virtual environment systems, therefore, are unique in that they can generate reconfigurable multimodal sensorimotor near-field interactions. A large and growing percentage of the tasks encountered in military, commercial, and private enterprises, however, are predominantly cognitive in nature. For example, many of these tasks require mainly that the human operator control and monitor a system through the use of a control panel. Although the main interaction with such control panels is through the sensorimotor system, the requirement to know what controls to operate and when to operate them is largely a cognitive task. One viewpoint is that, given the aforementioned trends, training for manipulative skills is unimportant. Strong proponents of this view argue that the majority of IMPORTANT skills, independent of frequency of occurrence, are largely cognitive in nature (Welford, 1976). By extension, it is argued that, since sensorimotor skills are not the main skills which must be trained, sensorimotor involvement in VE-based training systems is not required or is of secondary importance. In other words, it is assumed that sensorimotor involvement is not needed for the training of cognitive skills. In the research described in this thesis, these assumptions were tested and the effect of multimodal sensorimotor involvement in the training of predominantly cognitive tasks was examined in a task which requires subjects to: 1) identify a discrete system state, and 2) identify a series of discrete actions which must be taken in response to that state. Training effectiveness is measured and compared between 1) "classical" textbased training methods and 2) training methods which provide visual stimuli and which require haptic sensorimotor involvement. 2.0 Background Intellectual or cognitive skills "link perception and action and are concerned with translating perceptual input into a skilled response by using appropriate decisions." Colley and Beach, 1989, p. 2 From a training research viewpoint, VE-based systems are interesting only to the extent that training efficiency, skill performance, or skill retention are influenced and/or psychophysical insight may be gained. From a practical viewpoint, issues of cost (e.g., training system cost, time/cost required to train to a certain level of proficiency, and resource use costs), safety (i.e., limiting trainee exposure to threatening situations), portability (i.e., ability to effect training at a variety of sites) and reconfigurability (i.e., ability to use the same system to train a variety of skills) are of import. Lower training effectiveness measures may be tolerable in certain situations if the system's practical factors evaluate positively for a given training system. Nonetheless, a crucial factor in the evaluation of any training methodology or system is its training effectiveness. To date, however, research relevant to assessing the quantitative impact of multimodal sensorimotor involvement in the training of predominantly cognitive tasks has been minimal. Current models that are directed towards the learning of cognitive skills often fail to consider the possible effects of sensorimotor involvement in the learning of these cognitive skills (e.g., Anderson 1990, Posner, 1989; Preece, Rogers, Sharp, Benyon, Holland, and Carey, 1994; Benyon and Murray, 1993; Frederiksen and White, 1993; Roberts, 1993). Most relevant research has occurred in the use of subject performed tasks (SPTs) and experimenter performed tasks (EPTs) in event/action recall memory experiments (e.g., Cohen, 1981; Cohen, Peterson, and Mantini-Atkinson, 1987; Koriat, Ben-Zur, and Nussbaum, 1990; McAndrews and Milner, 1991). These experiments, however, explore the effects of multimodal sensorimotor involvement on the recall of tasks which are inherently sensorimotor in nature. To the knowledge of the author, principled examination of the impact of multimodal sensorimotor involvement on the training of predominantly cognitive skills is non-existent. Why pursue work in this area? First of all, insight into the role of multimodal sensorimotor involvement in training will have strong implications for the practical design of VE-based training systems (e.g., assessing if the added complexity and cost of haptic and auditory components is worthwhile for a given training task). Second, experimental results may aid in extending current cognitive training theory to encompass sensorimotor involvement. The question still remains, however, as to what evidence or plausible mechanisms exist which indicate that multimodal sensorimotor involvement may have an effect on the training of cognitive tasks? Changes in training performance, like changes in any cognitive process, can be brought about by effects on human perception, attention, or memory (Anderson 1990). Effective memory, in turn, may be promoted by: 1) organizing, relating, and searching for meaning in material; 2) forming a visual image of the material to be remembered; and 3) memorizing material in a manner which promotes cues for easier retrieval (Gazzaniga, 1988; Schacter, 1989; Bartlett, 1932; Paivio, 1971; Tulving and Donaldson, 1972; Tulving, 1983). Many of the potential mechanisms by which multimodal sensorimotor involvement may affect cognitive training involve one or more the these routes to promoting memory. The following are some, not necessarily independent, ways in which multimodal sensorimotor involvement may increase training effectiveness through effects on human perception, attention, and/or memory: 1. Dimensionality of Stimuli As discussed in Miller's classic paper (Miller, 1956), increased dimensionality of stimuli leads to increased span of absolute judgment and immediate memory. Multimodal interactions, where each sensory modality may be viewed as at least a single dimension, therefore, may potentially influence training through effects on perception and memory. 2. Context Isomorphisms A large body of research (e.g., see Anderson, 1990; Godden and Baddeley, 1975; Smith, Brown, and Toman, 1978) indicates that matching the training context to the recall context positively influences memory. Inasmuch as full multimodal feedback and sensorimotor involvement during training allows the training context to more closely approximate the recall or performance context, memory may influenced. 3. Depth or Elaborateness of Processing Memory research has shown that increasing the amount or elaborateness of mental processing during encoding increases recall (Anderson, 1990; Craik and Lockhart, 1972; Slamecka and Graf, 1978). Sensorimotor action has been shown to require significant cognitive processing (Jordan and Rosenbaum, 1989) and each modality which must be processed also increases the mental load (Anderson, 1990). It is possible, therefore, that adding multimodal sensorimotor involvement to cognitive tasks may influence recall. The previously mentioned SPT/EPT studies did precisely that and have shown increased event/action recall. Although the mechanisms are not clear, Koriat, Ben-Zur, and Nussbaum's (1990) work is particularly interesting. Subjects were given a series of verbal instructions (e.g., move the cup) and their recall was tested on the basis of expected and actual performance mode (i.e., verbal versus action). Subjects evidenced superior memory in the case that the performance mode was through action and in the case that it was EXPECTED that the performance mode would be through action. 4. Memory Cueing Perceptual-motor skills can be retained after very long periods of time without practice (Annett, 1989). This contrasts sharply with the retention of verbal material which may decline quite rapidly. A classic experiment showing the retention of perceptual motor skills was performed by Hill (1934, 1957). In this study, one day of typing practice was sufficient to achieve a level of skill which required 27 days of practice when originally learned 25 years in the past. If the cognitive components of a task could be related to a perceptual-motor skill in a robust manner, then multimodal sensorimotor involvement could influence cognitive skills training through memory mechanisms (i.e., by perceptual-motor memories providing strong cues to cognitive memories). Although the list of possible ways in which multimodal sensorimotor involvement may influence training can be further elaborated and organized, and issues concerning theory addressed more fully, clearly there is reason to suspect that multimodal sensorimotor involvement may have an effect on the training of cognitive skills. Key questions which must be answered concerning the role of multimodal sensorimotor involvement on cognitive skills training include: Can sensorimotor involvement (i.e., using a multimodal human/machine interface) aid in training cognitive skills and tasks? Which skills? Which tasks? What aspects of training are affected (i.e., efficiency, performance, and/or retention )? What is the relative impact of the various modalities? What are the key intramodal features? How can current theory be extended to account for multimodal effects? The research described in this subsequent sections of this thesis takes the smallest of steps along the path towards answering these questions. The EXISTENCE and EXTENT of the impact of multimodal sensorimotor involvement on training effectiveness are examined within the context of an explicit memory task.